Section: Research Program

Transverse Activity: Organisation of Challenges

Challenges have been an important drive for Machine Learning research for many years, and TAO members have played important roles in the organization of many such challenges: Michèle Sebag was head of the challenge programme in the Pascal European Network of Excellence (2005-2013); Isabelle Guyon, as mentioned, was the PI of many challenges ranging from causation challenges [69], to AutoML  [70]. The Higgs challenge  [55], most attended ever Kaggle challenge, was jointly organized by TAO (C. Germain), LAL-IN2P3 (B. Kegl) and I. Guyon (not yet at TAO), in collaboration with CERN and Imperial College. The challenge activity continue d within TAU, in relation with fundamental and applied issues.

TAU is particularly implicated with the ChaLearn Looking At People (LAP) challenge series in computer vision, in collaboration with the University of Barcelona [46]. Notably in 2017, TAU co-organized several international LaP challenges:

• ChaLearn Looking at People (LAP) - Job Candidate Screening Coopetition [21]. In conjunction with IJCNN 2017 workshop on explainability in machine learning.

• ChaLearn LAP Real Versus Fake Expressed Emotion Challenge (ICCV 2017) [30].

• ChaLearn LAP Large-scale Continuous Gesture Recognition Challenge (ICCV 2017) [30].

• ChaLearn LAP Large-scale Isolated Gesture Recognition Challenge (ICCV 2017) [30].

TAU was also implicated in organizing a follow up of the AutoML challenge for the PAKDD conference. TAU also co-organized local events (hackathons), as “rehearsals’’ of international competitions in preparation:

• Spatio-temporal time series challenges for the European See.4C challenge about Energy Management (Paris, 14/2/2017, and Toulon, 22/4/2017). Book with Sprimger in preparation.

• Track ML: tracking particles in high energy physics (Orsay, 21/3/2017) [16].

The Codalab challenge platform, originally designed within Microsoft Research with Isabelle Guyon as one of the PIs, has now been migrated to U. Paris-Sud. It is an open source project. Part of the development is supported by Isabelle Guyon’s Paris-Saclay chair (co-funded by Inria). Codalab’s user base has been steadily growing. At the end of 2017, we now have over 10’000 users who have entered more than 480 challenges (145 of which are public).

This year, there was a major upgrade of Codalab, featuring:

• A step-by-step Wizard to guide beginner challenge organizers through the process of organizing challenges. This Wizard facilitates the work of students learning to organize challenges.

• Use of dockers and queues, allowing challenge participants to easily use their own computer resources in the backend to support challenges with code submissions.

• A modular competition logic, which will enable supporting new types of challenges such as reinforcement learning competitions.